The IRS sent notices in March to an estimated 20 million recipients of Social Security and some veteran's benefits, who were eligible to receive stimulus checks, said Shulman. Only about 15 million of them have filed returns or are listed on somebody else's return as a dependent.

As of June 13, the agency had sent out around $63.8 billion to 76 million Americans, according to Shulman. The IRS is on track to send out a total of 110 million payments by July 11. However, applying for benefits requires that individuals file an income tax return.

People whose main income is Social Security or veterans benefits may not be required to file a federal tax return.

"Communicating with this population presents a significant challenge, given the complications of age and disability:' Nina Olson, National Taxpayer Advocate, told the committee.

Olson cited lack of mobility, lack of access to or familiarity with the Internet, confusing information and fear of contact with the IRS as possible barriers to filing.

The IRS said it would continue to work with other organizations as it reaches out to veteran and retiree communities through the summer and helps coordinate free face-to-face tax preparation services for them.