As the daughter of the artist Broncia Koller-Pinell and the industrialist Hugo Koller, Silvia Koller grew up on the family estate in Oberwaltersdorf near Baden, which was a popular meeting place for artists. As early as 1910, she attended Franz Čižek's famous drawing courses. From 1914 to 1918, Koller studied at the School of Applied Arts in Vienna with Adolf Böhm, Rudolf von Larisch and Koloman Moser. In 1921 Koller moved to Berlin and continued her studies with Carl Hofer. In 1924, she went to Geneva to see Alexandre Blanchet and from there to Paris. In 1937, Koller returned to her parents' home in Oberwaltersdorf to care for her sick father and subsequently withdrew from the art world. It was not until 1953 that she took part in courses at the Salzburg Summer School, which were led by Oskar Kokoschka. Koller's preferred subjects were landscapes, vedute, portraits, still lifes and animal depictions.