undefined | Better HN

0 points_hark1y ago0 comments

Totally fair points all. Sorry if it came across as condescending!

I agree with you that this network probably has not found the source code or something like a minimal description in its weights.

Honestly, I'm writing a paper on model compression/complexity right now, so I may have co-opted the discussion to practice talking about these things...! Just a bit over-eager (,,>﹏<,,)

Have you given much thought to how we can encourage models to be more compressible? I'd love to be able to explicitly penalize the filesize during training, but in some usefully learnable way. Proxies like weight norm penalties have problems in the limit.

0 comments

godelski1y ago

Haha totally fair and it happens to me too, but trying to work on it.

I actually have some stuff I'm working on in that area that is having some success. I do need to extend it to diffusion but I see nothing stopping me.

Personally I think a major slowdown for our community is it's avoidance of math. Like you don't need to have tons of math in the papers, but many of the lessons you learn in the higher level topics do translate to usable techniques in ML. Though I would also like to see a stronger push on theory because empirical results can be deceiving (Von Neumann's elephant and all)

j / k navigate · click thread line to collapse

0 comments

godelski1y ago

Haha totally fair and it happens to me too, but trying to work on it.

I actually have some stuff I'm working on in that area that is having some success. I do need to extend it to diffusion but I see nothing stopping me.

j / k navigate · click thread line to collapse