This "How to Read the Early Christian Writings" was orginally the end of the Polycarp page. So it's main reference is Polycarp's letter to the Philippians. I haven't changed anything, because Polycarp's letter definitely qualifies as a good first to read from the early Church fathers.
Don't just read them! Experience them!
You'll have to do that with several other letters to get the full effect. Christianity—real, powerful, life-changing, fearless Christianity—is better caught than taught. You'll never simply study Christianity and know how to follow Christ.
That's why the Bible, especially the New Testament, is worth reading over and over. We know from history that you can find the whole Gospel in the Old Testament, but the New Testament is where you get to feel the way God would live if he were human. First he showed us through his Son, as one man, Jesus Christ, and then he shows us through the church, the corporate body of Christ.
You can see the life of Christ in a gathered people in Acts and in the apostles' writings, but that continues on in the churches the apostles started. Men like Polycarp, and later men like Irenaeus, Clement of Alexandria, Origen—and even the bitingly sarcastic Tertullian—give us a feel, more than a teaching, about how Christ lives through men.
Irenaeus described Clement of Rome, whose First Clement is a longer but also excellent read, in words that describe the way pure Christianity is felt and experienced:
Polycarp's letter is an excellent place to begin that feeling and experiencing.
If you want to read the early Christian writings for free, the best way is to Google them. That will give you a choice of translations, rather than being stuck with some 1860's version. Pretty much everything is available online.
If you prefer paper to read rather than finding it online, Penguin's The Apostolic Fathers is the cheapest way to get a few very early and excellent Christian writings.
And, of course, the standard best collection of all the Pre-Nicene writings is The Ante-Nicene Fathers, a 10-volume set. That's all translations made in the late 1800's, but it is everything in one place. There's some excellent notes and introductions as well, even if they're over a century out-of-date. (It's kind of fun reading the occasional sarcastic comments by the American editor, who was a lot less friendly to Catholicism than the original Edinburgh, Scotland editors.)