Hi,

I’d like to explore the possibilities of training a LM to learn a specific programming language so he can be used as co-pilot in that context. Language is a niche language (http://pharo.org), and there is no existing model nowadays knowing it (also, I want to make some extra tweaks, once I have it).
Thing is… I have no idea where to start! :)

Any hint where can I learn the ropes?

Thanks!

  • j4k3@lemmy.world
    link
    fedilink
    English
    arrow-up
    4
    ·
    1 year ago

    It’s going to seem so wrong, but… stable diffusion and porn is the easiest way to figure out the basics of modifying a model. It is easier to spot mistakes and the number of examples to try is enormous. This space is packed with examples running on basic low level hardware and you’ll find lots of easy to follow examples. The bar is much higher when it comes to training examples for text or code. Most examples are based on proprietary toolchains or are very basic getting started guides.

    • abhibeckert@lemmy.world
      link
      fedilink
      English
      arrow-up
      3
      arrow-down
      2
      ·
      edit-2
      1 year ago

      … stable diffusion is a diffusion model - OP wants a language model. They don’t work even remotely the same way.

      • rufus@discuss.tchncs.de
        link
        fedilink
        English
        arrow-up
        1
        ·
        edit-2
        1 year ago

        I’d agree. And the toolchains aren’t proprietary. But they are different and you can transfer only a small amount of knowledge from one to the other. The concept ‘training an AI model’ is the same. PyTorch, Linux and the graphics card are the same. So you can lean something. But it’s another kind of AI model and the software stack you need to use also differs.