On weekdays, every 5 minutes between 11:30 and 16:45
DTSTART:20040101T000000
RRULE:FREQ=DAILY;COUNT=100;BYDAY=MO,TU,WE,TH,FR;BYHOUR=11,12,13,14,15,16;BYMINUTE=30,35,40,45
EXDATE:20040101T000000
To explain:
FREQ=DAILY because the pattern "repats" every day.
BYDAY=MO,TU,WE,TH,FR limits the days on which the pattern works to weekdays.
BYHOUR=11,12,13,14,15,16 "expands" the pattern to work on these hours. It "expands" the pattern because otherwise a daily pattern would normally fire only once.
BYMINUTE=30,35,40,45 "expands" the pattern to work on these minutes only.
All of the above gives you the combination you want.
A pattern has to have a DTSTART. According to the iCalendar standard, the start date will be included in the resulset. You probably don't want it, so it is excluded from the result set using EXDATE.