Runnin llm's with ollama on SFOS

I found out that ollama has arm64 binaries: Release v0.1.42 路 ollama/ollama 路 GitHub

So I thought its worth a try.
I downloaded the binary, ran chmod +x Downloads/ollama-linux-arm64 and Downloads/ollama-linux-arm64 serve and then you can actually run models in another terminal session. (Downloads/ollama-linux-arm64 run llama3 "tell the history of linux on mobile phones")
I tried llama3, its quite slow, maybe a word every 15 seconds on my 10iii, but it works :slight_smile:

2 Likes

:slight_smile:

How much memory does it take? I鈥檓 guessing OOM problems aren鈥檛 exactly helped by doing that right?

A 4.7 GB download鈥 :grin:

Didnt check, but used the phone while it was genrating and it wasnt worse than else

Yes, also didnt check where it is downloaded to. Might also be worth checking the other models available with ollama

Trying phi3 right now, on my desktop.

And here鈥檚 your answer from phi3鈥 :wink:

Summary

$ ollama run phi3 鈥渢ell the history of linux on mobile phones鈥
pulling manifest
pulling b26e6713dc74鈥 100% 鈻曗枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻 2.4 GB
pulling fa8235e5b48f鈥 100% 鈻曗枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻 1.1 KB
pulling 542b217f179c鈥 100% 鈻曗枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻 148 B
pulling 8dde1baf1db0鈥 100% 鈻曗枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻 78 B
pulling f91db7a2deb9鈥 100% 鈻曗枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻 485 B
verifying sha256 digest
writing manifest
removing any unused layers
success
Linux has a unique and evolving relationship with mobile phones, which began in
earnest with the advent of smartphones. Its journey into this market is marked by both
challenges and successes, as open-source developers aimed to provide alternatives to
proprietary operating systems like Android (based on Google鈥檚 own version of Linux).
Here鈥檚 an overview of Linux鈥檚 history within mobile phones:

Early Beginnings: PDAs & Palm OS/Linux

Before smartphones became ubiquitous, there were Personal Digital Assistants (PDAs),
which served as early forms of mobile computing devices. In the late 1980s and 1990s,
Linux was used to develop alternative operating systems for PDAs. One example is Palm
OS, an object-oriented real-time operating system designed for synchronizing personal
information with a handheld device.

Smartphones & Android: The Dominance of Proprietary Systems

The first smartphone was the IBM Simon Personal Communicator, released in 1994.
However, it wasn鈥檛 until the release of Apple鈥檚 iPhone and its subsequent operating
system iOS that mobile computing became widely popular. At this time, Android started
to emerge as a significant Linux-based alternative with an open-source kernel called
鈥淟inux for Mobile.鈥

Android: Opening Up Possibilities

In 2005, Google acquired the company Android Inc., which was developing its own mobile
operating system. In 2007, they launched Android as a Linux distribution tailored for
smartphones and tablets, with open-source components that allowed manufacturers to
customize it according to their preferences while maintaining compatibility across
devices.

Linux Distributions in Mobile Devices: Post-Android Era

While Android dominates the mobile market today, other Linux distributions have also
made their way into smartphones and tablets, albeit less prevalently:

  1. Tizen OS - Developed by Samsung Electronics, Tizen is a Linux-based operating
    system designed for smartphones, tablets, wearables, cameras, and smart TVs. It鈥檚 an
    open-source project that competes with Android in some markets.
  2. Maemo/MeeGo - Once used by Nokia for its high-end smartphones (such as the
    N900), Maemo was a Linux distribution based on Debian. MeeGo is an open-source
    operating system that succeeded Maemo, but it didn鈥檛 achieve significant market
    penetration and eventually merged into Tizen in 2013.
  3. Ubuntu Touch - Developed by Canonical Ltd., Ubuntu Touch was initially aimed at
    smartphones with Nexus hardware partnership until its development slowed down after the
    acquisition of Blackberry鈥檚 assets, which included their QNX operating system for
    mobile devices. However, in recent years, there has been a resurgence in interest
    within the community to develop an open-source version of Ubuntu Touch specifically for
    smartphones and tablets.
  4. Firefox OS - Based on the popular web browser Firefox, this Linux distribution
    aimed at providing a mobile operating system that runs applications directly from the
    Web browser. While it failed to gain widespread adoption, its principles have
    influenced other projects such as Sailfish OS (created by Jolla) and Ubuntu Touch鈥檚
    approach to app development.

Challenges & Future Prospects

Linux on mobile phones has faced challenges due to the dominance of proprietary
operating systems like Android and iOS, which benefit from significant investments in
user-friendly interfaces, developer ecosystems, and strong brand identities. However,
open-source communities continue to innovate and explore new possibilities for Linux
distributions on mobile devices.

As smartphones evolve with the proliferation of IoT (Internet of Things) devices and
wearable technology, it鈥檚 likely that more customizable, secure, and privacy-focused
options powered by open-source systems such as Linux may find their place in this
expanding market.

I thought I should try the smallest model ( moondream) on my X10III, but I can鈥檛 even get through the model download. The binary (ollama-linux-arm64) constantly crashes after a few MB, saying it is searching for a better connection. Is there a speed requirement for the connection?
Living on the country side my 4G connection isn鈥檛 much to brag about. :unamused:

Likely, the post will be used to train another version of ollama, chat-gpt and others.

1 Like

You could try to download from huggingface and just place it where ollama expects it (is it this one? moondream/moondream2-gguf at main)

2 Likes

I think it is.

The model may generate offensive, inappropriate, or hurtful content if it is prompted to do so. :smile:

鈥渁 tiny vision language model that kicks ass and runs anywhere鈥

Here is ollama built on device (XIII) https://files.catbox.moe/6c0l6n.gz maybe it will be less crashy? Seems to continue downloads at least for me


Edit: yeah I鈥檝e no clue how to feed it images and oom is an issue, so might need extra swap, stopping appsupport etc, but download seemed to work (but I鈥檓 in a middle of a city so no idea)
And yeah downloading by hand will be tricky, this is what it downloaded in ~/.ollama/

Edit2: oh wow it works took around 2 minutes but:

Picture in question:

(also totally nuked network connectivity, blank top right corner, utilities restart useless as always, airplane mode, no diffie, full restart required, maybe on XV (8Gb) worth revisiting)

Thanks! I鈥檒l take a look at it tomorrow. My spare time is out for today. :wink:

I had no problems with crashes and smooth downloads, sounds like a connection issue

Uh, nice. I wonder if there are a few easy to get speedups by building ollama for sfos and setting a few configs

Yes it is a connection issue. Also @throwaway69鈥檚 build is complaining about it and crash.
Well, I鈥檝e known for a long time that I should re-flash my X10III, but as it is my one and only daily drive with some critical apps, I am waiting for the X10V image or the C2, whichever comes first.
So no AI on my phone for now.

Btw鈥 utilities restart worked fine for me.

1 Like

A bit OT, but I just bought another 32 GB RAM stick for my PC, to play with these things. I doubt it, but maybe I鈥檒l find some real use for it. :wink:

A bit off topic, but currently hacking on this, so wanted to share.

Needs about 200MB. :slight_smile:

SCNR.

3 Likes