The problem is string comcatenation is using the same operator as addition:
+
. If one of the operands is a number SQL server assumes it's addition and tries to convert the rest to numbers too. So what you need is to convert result of count to varchar:
select 'INSERT INTO CM_ONTHEFLYCOMMUNICATION (c_source,c_classification,c_name,c_email,c_mobile,c_bookscount)
values ( ''Library'','''+isnull('CheckOut -'+cio_branchname,'')+''',
'''+ISNULL(cio_membername,'')+''','''+ISNULL(mstmember.m_email1,'')+''',
'''+ISNULL(mstmember.m_mobile,'')+''','''+cast(count(trncheckinout.cio_bookname) as varchar(100))+''')'
AS MAILQRY
from
trncheckinout inner join
mstmember on cio_mrecid=m_recid where CIO_CheckOutDt = 20141208
group by m_email1,m_mobile, cio_branchname,cio_membername
I took the liberty to remove the concatenation around the first operand 'Library'.