How far back does you calendar go? I had the same problem and found that by enabling "delete events after 90 days," (in iCal prefs) there was a significant improvement in performance. Another trick is to limit the number of subscribed calendars. I subscribe and publish several, but I don't always need to see all of them displayed at once. If this applies to you, just pick one or two key calendars and uncheck the rest (you can view them later when needed simply by rechecking the box).
P.S. Mine loads in about 5 seconds, running on a 1 ghz G4 with 512 megs RAM, but my old iMac 400 (same amount of RAM as the G4) isn't far behind it, loading under 10 seconds.