We report on a search for gravitational waves from the coalescence of compact binaries during the third and fourth LIGO science runs. The search focused on gravitational waves generated during the inspiral phase of the binary evolution. In our analysis, we considered three categories of compact binary systems, ordered by mass: (i) primordial black hole binaries with masses in the range 0.35M⊙<m_1, m_2<1.0M⊙, (ii) binary neutron stars with masses in the range 1.0M⊙<m_1, m_2<3.0M⊙, and (iii) binary black holes with masses in the range 3.0M⊙<m_1, m_2<m_maxwith the additional constraint m_1+m_2<m_max, where m_max was set to 40.0M⊙ and 80.0M⊙in the third and fourth science runs, respectively. Although the detectors could probe to distances as far as tens of Mpc, no gravitational-wave signals were identified in the 1364 hours of data we analyzed. Assuming a binary population with a Gaussian distribution around 0.75−0.75M⊙, 1.4−1.4M⊙, and 5.0−5.0M⊙, we derived 90%-confidence upper limit rates of 4.9 yr^-1 L_10^-1for primordial black hole binaries, 1.2yr^-1 L_10^-1for binary neutron stars, and 0.5yr^-1 L_10^-1for stellar mass binary black holes, where L_10 is 10^10 times the blue-light luminosity of the Sun.