I think your analysis of the damage being cause by failing gallery gaskets is probably spot on. Your question of whether its even worth the tear down is the real kicker. When I did the gaskets I also put in a CJM oil pan; I found bits of gasket material in my oil pan but no metal shavings. To see that amount in the drain pain would definitely be concerning to me. Other more experienced individuals may have better insight.
You need to have in your mind what your bottom line is. If you like the car and have the funds available, then it may be worth a tear down to check the extent of the damage. Otherwise, you may need to be prepared to bail
