Blog

Streaming data with spring boot restful web service examples here, illustrate different ways to stream data like stream JSON, stream CSV, stream chunked response, stream large file as zip file, dynamically created file, static files from resource folder, stream video, stream audio etc. We will build spring boot rest api examples for all of these types of content.

Streaming data with spring boot restful web services is used for asynchronous request processing as a non blocking operation, application directly writes the content to output response stream. Client keeps alive connection with application server and keeps on receiving response stream, until application sends content’s last chunk in response and confirms the completion of response stream.

Spring Boot REST api streaming Options

1. HttpServletResponse’s OutputStream – old approach

This is the oldest approach being used. In this approach OutputStream is obtained from HttpServletResponse and then content is written to the OutputStream object. We will not use this approach in the upcoming examples.

2. StreamingResponseBody as return type

Spring Boot Rest api streaming with StreamingResponseBody is the most easiest and elegant way to create a rest web service to stream content. I generally prefer to use StreamingResponseBody with ResponseEntity, so we can set the headers and HTTP status, in this way we can control the behavior of our API in a much better way.

StreamingResponseBody type is used for async request processing and content can be written directly to the response OutputStream without holding up the threads in the servlet container.

StreamingResponseBody timeout should be increased if you are returning a huge stream and your application throws a timeout exception. Following property can be set in application.properties file or yml file to increase timeout.

spring.mvc.async.request-timeout = 3600000

3. Spring WebFlux publishers

Spring WebFlux is introduced in the Spring 5 version and works similar to Spring MVC. Spring WebFlux supports fully non blocking reactive streams and provides awesome features. Flux publisher present in Spring WebFlux provides ready-made support for streaming data, without providing any additional configuration.

Project creation and setup

Create a new Spring Boot project using Maven or Gradle. I will prefer to use Java 11 and Gradle based projects for explaining all the examples. Open Spring Initializr and create a new project. Add two dependencies web and webflux in the project. 

I have created a sample project and my project structure looks like this, this project structure will be used in all examples to explain different functionalities. Don’t worry if your project is empty as of now, we will create each and every class in the project.

1. Streaming data with Spring Boot RESTful web service example

Spring Boot REST API example to stream content will be created here. First of all create a controller, in our example APIController class is created for this purpose. APIContoller class is annotated with @RestController annotation, which represents that all the endpoints will be REST API and no view resolver is required for this class. 

1.1 Stream data example using StreamingResponseBody 

In APIContoller, create a method for our REST API with ResponseEntity<StreamingResponseBody> as return type. We are using ResponseEntity here which helps in better control of HTTP headers and status. Following is the sample code to create streaming data from Spring Boot REST API.

@RequestMapping("/api/stream")
@RestController
public class APIController {

  @GetMapping(value="/data")
  public ResponseEntity<StreamingResponseBody> streamData() {
     StreamingResponseBody responseBody = response -> {
        for (int i = 1; i <= 1000; i++) {
           try {
              Thread.sleep(10);
              response.write(("Data stream line - " + i + "\n").getBytes());
           } catch (InterruptedException e) {
              e.printStackTrace();
           }
        }
     };
     return ResponseEntity.ok()
           .contentType(MediaType.TEXT_PLAIN)
           .body(responseBody);
  }
}

In this example, we are simulating the content with a loop to generate multiple strings to send as a stream response. I have added a sleep method so that you can see & identify the streaming response. At the end of the REST API method add ResonseEntity with HTTP status OK (200) and contentType as plain text. StreamingResponseBody will be added into the body section of ResponseEntity. Start the server and add following url in the web browser

http://127.0.0.1:8080/api/stream/data

1.2 Stream data example using WebFlux 

Flux has the streaming capabilities out of the box. The whole code mentioned in the previous example can be replaced with a single line and everything works exactly in the same way.

Create a REST endpoint method with return type as Flux<Object> and in mapping mention that it will generate a streaming JSON or octet values (as we only have 2 streaming values, which can be used for normal content also).

@GetMapping(value = "/data/flux", produces = MediaType.APPLICATION_STREAM_JSON_VALUE)
public Flux<Object> streamDataFlux() {
  return Flux.interval(Duration.ofSeconds(1)).map(i -> "Data stream line - " + i );
}

After creating the above REST endpoint, open the browser and add the following url:

http://127.0.0.1:8080/api/stream/data/flux

1.3 Stream data example output

You will see the streaming output in the browser as shown in the following image, both ways will show the exact same output.

2. Streaming JSON with Spring Boot RESTful web service example

Spring Boot REST apis have the ability to stream JSON objects also. Streaming JSON is somewhat different from traditional REST based API with JSON response. In normal REST api, there is a parent JSON object under which children are added as an array to send a long list of JSON objects. But in JSON streaming, there is no parent JSON object, as each object is itself a complete json object and every JSON object is streaming individually. It will be more clear with the following example, where we see how to stream a large JSON via Spring Boot REST api.

2.1 Stream JSON example using StreamingResponseBody 

In APIController, create a REST api endpoint using a method which has a return type as ResponseEntity<StreamingResponseBody> and content type in ResponseEntity will be APPLICATION_STREAM_JSON. This content type tells the client that it will stream json objects, so keep the connection alive until the server confirms the end of the response stream.

In the following example, I am using Jackson’s ObjectMapper to convert the object to JSON. The JSON object as a string will be written to the object of StreamingResponseBody and to simulate streaming we have added a thread sleep of 1 second.

@GetMapping("/json")
public ResponseEntity<StreamingResponseBody> streamJson() {
  int maxRecords = 1000;
  StreamingResponseBody responseBody = response -> {
     for (int i = 1; i <= maxRecords; i++) {
        Student st = new Student("Name" + i, i);
        ObjectMapper mapper = new ObjectMapper();
        String jsonString = mapper.writeValueAsString(st) +"\n";
        response.write(jsonString.getBytes());
        response.flush();
        try {
           Thread.sleep(1000);
        } catch (InterruptedException e) {
           e.printStackTrace();
        }
     }
  };
  return ResponseEntity.ok()
        .contentType(MediaType.APPLICATION_STREAM_JSON)
        .body(responseBody);
}

Open the browser and enter the following url, to see the streaming JSON:

http://127.0.0.1:8080/api/stream/json

2.2 Stream JSON example using WebFlux 

Using WebFlux’s Flux publisher, you can directly mention the class type with Flux as Flux<ClassName> as a return type of REST endpoint method. In our example, we have a method with return type as Flux<Student> to return the stream of Student objects .

@GetMapping(value = "/json/flux", produces = MediaType.APPLICATION_STREAM_JSON_VALUE)
public Flux<Student> streamJsonObjects() {
  return Flux.interval(Duration.ofSeconds(1)).map(i -> new Student("Name" + i, i.intValue()));
}

In the illustrated example, I am simulating the content using a loop via interval. But if you have configured the database, then you can mention the repository’s method directly which will return a list of objects. Flux automatically manages the database cursors, so for a large dataset, it streams all the database objects without overloading the server resources.

@GetMapping(value = "/json/flux", produces = MediaType.APPLICATION_STREAM_JSON_VALUE)
public Flux<Student> streamJsonObjects() {
  return studentRepository.findAll();
}

In web browser, enter the following url to see the streaming json content

http://127.0.0.1:8080/api/stream/json/flux

2.3 Stream JSON example output

In above both examples, you will see the same output in a web browser like this.

3. Streaming dynamically created file with Spring Boot RESTful web service  example

In this example, we will create a stream of text which will be downloaded as a file on the client side. The special thing here will be the file will start downloading immediately on the client side, but our streaming response will keep on writing the content in the downloaded file. Until the streaming response write is not complete from the server, the Web browser will keep showing the downloading of the file.

In APIContoller, create a new method for REST API which returns ResponseEntity<StreamingResponseBody> type. The special thing we will be doing here is about headers. In ResponseEntity, we will set a header for CONTENT_DISPOSITION which will have value as attachment and a filename. Based upon this, the client of REST API will identify that it should be downloaded as a file with mentioned filename.

@GetMapping("/textfile")
public ResponseEntity<StreamingResponseBody> streamContentAsFile() {
  StreamingResponseBody responseBody = response -> {
     for (int i = 1; i <= 1000; i++) {
        response.write(("Data stream line - " + i + "\n").getBytes());
        response.flush();
        try {
           Thread.sleep(10);
        } catch (InterruptedException e) {
           e.printStackTrace();
        }
     }
  };
  return ResponseEntity.ok()
        .header(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=test_data.txt")
        .contentType(MediaType.APPLICATION_OCTET_STREAM)
        .body(responseBody);
}

Once you created the above REST API, then add the following url in the web browser.

http://127.0.0.1:8080/api/stream/textfile 

You will see a file immediately get downloaded and the streaming response keeps on writing on that file.

4. Streaming file with Spring Boot RESTful web service example

In Spring Boot REST api, a very common requirement is to download existing files, which might be present in a file system, resources folder or on cloud like S3 etc. Let’s take an example of downloading a pdf file from the resource folder.

In APIContoller, create a new method for REST api endpoint. Here we will simply create an object of StreamingResponseBody and provide the path of the file to be downloaded. The REST of work will be done by StreamingResponseBody automatically to stream the content of the file. In ResponseEntity, we need to mention the content type and content disposition header. Using headers, the client can clearly understand the type of content will be coming and how to treat it whether to download or not etc.

@GetMapping("/pdfFile")
public ResponseEntity<StreamingResponseBody> streamPdfFile() throws FileNotFoundException {
  String fileName = "Technicalsand.com sample data.pdf";
  File file = ResourceUtils.getFile("classpath:static/" + fileName);
  StreamingResponseBody responseBody = outputStream -> {
     Files.copy(file.toPath(), outputStream);
  };
  return ResponseEntity.ok()
        .header(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=Downloaded_" + fileName)
        .contentType(MediaType.APPLICATION_PDF)
        .body(responseBody);
}

Once above code is added into your application, start the application. Open the web browser and add the following url.

http://127.0.0.1:8080/api/stream/pdfFile

You will see a PDF file will be downloaded as a streamed file.

5. Streaming csv with Spring Boot RESTful web service example

Downloading a dynamic CSV file as stream is very similar to downloading a text file as stream, as discussed in earlier examples. But for the sake of clarity, let’s build this example separately. In this example, the content written on StreamingResponseBody will be written as comma or tab separated values for creating a valid CSV row. In the header section, the filename will have the extension as .csv. Other things will remain the same as build in earlier examples.

@GetMapping(value = "/csv")
public ResponseEntity<StreamingResponseBody> getCsvFile() {
  StreamingResponseBody stream = output -> {
     Writer writer = new BufferedWriter(new OutputStreamWriter(output));
     writer.write("name,rollNo"+"\n");
     for (int i = 1; i <= 10000; i++) {
        Student st = new Student("Name" + i, i);
        writer.write(st.getName() + "," + st.getRollNo() + "\n");
        writer.flush();
     }
  };
  return ResponseEntity.ok()
        .header(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=data.csv")
        .contentType(MediaType.TEXT_PLAIN)
        .body(stream);
}

Start the application and in the web browser, add the following url.

http://127.0.0.1:8080/api/stream/csv 

A CSV file will be downloaded and streaming content will keep on writing in the file, until server response is not complete.

6. Streaming large data as zipped file from REST Services example

Downloading a big file as a ZIP file or multiple files as a single ZIP file is the most efficient way to transfer files. Spring provides out of the box support for creating dynamic ZIP files and writing content on the files inside the zip file as stream. This means, if you have created a ZIPPED file and a file added inside still can accept the streaming content.

Create a REST api method in APIContoller, as shown in the code snippet here. The content disposition here will have a file name with extension zip. The method writeToStream does all the magic. It creates an object of ZipOutputStream using BufferedOutputStream, which is further internally using StreamingResponseBody. 

First of all, you need to create a new file entry inside ZipOutputStream using ZipEntry, in given example its “data.csv” file which will be present inside the ZIP file. Now create a BufferedWriter for ZipEntry using OutputStreamWriter which internally uses ZipOutputStream. Now keep on writing the content on the BufferedWriter object.

@GetMapping(value = "/zip")
public ResponseEntity<StreamingResponseBody> getZipFileStream() {
  StreamingResponseBody stream = output -> writeToStream(output);
  return ResponseEntity.ok()
        .header(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=report.zip")
        .contentType(MediaType.APPLICATION_OCTET_STREAM)
        .body(stream);
}

public void writeToStream(OutputStream os) throws IOException {
  ZipOutputStream zipOut = new ZipOutputStream(new BufferedOutputStream(os));
  ZipEntry e = new ZipEntry("data.csv");
  zipOut.putNextEntry(e);
  Writer writer = new BufferedWriter(new OutputStreamWriter(zipOut, Charset.forName("UTF-8").newEncoder()));
  for (int i = 1; i <= 1000; i++) {
     Student st = new Student("Name" + i, i);
     writer.write(st.getName() + "," + st.getRollNo() + "\n");
     writer.flush();
  }
  if (writer != null) {
     try {
        writer.flush();
        writer.close();
     } catch (IOException e1) {
        e1.printStackTrace();
     }
  }
}

Once you have added the above code, start the application and add the following url in the web browser.

http://127.0.0.1:8080/api/stream/zip 

You will see a ZIP file immediately start downloading as a stream file. Once the file download completes, open the ZIP file and you will see a “data.csv” file present inside it.

7. Streaming audio/video with Spring Boot REST api example example

Spring Boot REST apis can be used to stream audio or videos.The important role is of headers , using which client and application communicates mutually and transfers the chunk of bytes, which are required. We will be using 4 HTTP headers in this example, following is the explanation of these headers, so that it becomes clear why we need to use these:

Content-Type	The Content-Type entity header is required to point to the media form of the resource. In responses, a Content-Type header tells the client what the content kind of the returned content actually is.
Accept-Ranges	The Accept-Ranges response header is a marker utilized by the server to inform its support for partial requests. This field’s value indicates the unit which will be used to define a range.
Content-Length	The header Content-Length is a number value representing the exact byte length of the HTTP response body.
Content-Range	The HTTP response header Content-Range indicates that the partial message belongs to full body massage at which part.

In Following code snippet, you can see we have used exactly the same private method to handle both audio and video content. Because both are binary files, and both work in the exact same way by sending the chunk of bytes, requested by the client.

@RestController
@RequestMapping("/audiovideo")
public class AudioVideoController {
  public static final String VIDEO_PATH = "/static/videos";
  public static final String AUDIO_PATH = "/static/audios";

  public static final int BYTE_RANGE = 128; // increase the byterange from here

  @GetMapping("/videos/{fileName}")
  public Mono<ResponseEntity<byte[]>> streamVideo(@RequestHeader(value = "Range", required = false) String httpRangeList,
                                      @PathVariable("fileName") String fileName) {
     return Mono.just(getContent(VIDEO_PATH, fileName, httpRangeList, "video"));
  }

  @GetMapping("/audios/{fileName}")
  public Mono<ResponseEntity<byte[]>> streamAudio(@RequestHeader(value = "Range", required = false) String httpRangeList,
                                      @PathVariable("fileName") String fileName) {
     return Mono.just(getContent(AUDIO_PATH, fileName, httpRangeList, "audio"));
  }

  private ResponseEntity<byte[]> getContent(String location, String fileName, String range, String contentTypePrefix) {
     long rangeStart = 0;
     long rangeEnd;
     byte[] data;
     Long fileSize;
     String fileType = fileName.substring(fileName.lastIndexOf(".")+1);
     try {
        fileSize = Optional.ofNullable(fileName)
              .map(file -> Paths.get(getFilePath(location), file))
              .map(this::sizeFromFile)
              .orElse(0L);
        if (range == null) {
           return ResponseEntity.status(HttpStatus.OK)
                 .header("Content-Type", contentTypePrefix+"/" + fileType)
                 .header("Content-Length", String.valueOf(fileSize))
                 .body(readByteRange(location, fileName, rangeStart, fileSize - 1));
        }
        String[] ranges = range.split("-");
        rangeStart = Long.parseLong(ranges[0].substring(6));
        if (ranges.length > 1) {
           rangeEnd = Long.parseLong(ranges[1]);
        } else {
           rangeEnd = fileSize - 1;
        }
        if (fileSize < rangeEnd) {
           rangeEnd = fileSize - 1;
        }
        data = readByteRange(location, fileName, rangeStart, rangeEnd);
     } catch (IOException e) {
        e.printStackTrace();
        return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).build();
     }
     String contentLength = String.valueOf((rangeEnd - rangeStart) + 1);
     return ResponseEntity.status(HttpStatus.PARTIAL_CONTENT)
           .header("Content-Type", contentTypePrefix
"/" + fileType)
           .header("Accept-Ranges", "bytes")
           .header("Content-Length", contentLength)
           .header("Content-Range", "bytes" + " " + rangeStart + "-" + rangeEnd + "/" + fileSize)
           .body(data);
  }

  public byte[] readByteRange(String location, String filename, long start, long end) throws IOException {
     Path path = Paths.get(getFilePath(location), filename);
     try (InputStream inputStream = (Files.newInputStream(path));
         ByteArrayOutputStream bufferedOutputStream = new ByteArrayOutputStream()) {
        byte[] data = new byte[BYTE_RANGE];
        int nRead;
        while ((nRead = inputStream.read(data, 0, data.length)) != -1) {
           bufferedOutputStream.write(data, 0, nRead);
        }
        bufferedOutputStream.flush();
        byte[] result = new byte[(int) (end - start) + 1];
        System.arraycopy(bufferedOutputStream.toByteArray(), (int) start, result, 0, result.length);
        return result;
     }
  }

  private String getFilePath(String location) {
     URL url = this.getClass().getResource(location);
     return new File(url.getFile()).getAbsolutePath();
  }

  private Long sizeFromFile(Path path) {
     try {
        return Files.size(path);
     } catch (IOException ex) {
        ex.printStackTrace();
     }
     return 0L;
  }
}

Now open the browser and add the following two urls one by one. 

http://127.0.0.1:8080/audiovideo/audios/audio1.mp3
http://127.0.0.1:8080/audiovideo/videos/video1.mp4

You will see your browser has started to stream both video and audio automatically.

8. Streaming audio/video with Spring Boot REST API example using Spring content example

This is another simple way to stream your media files. We will be using the Spring content project here, which provides support to stream media files present within applications as well as outside (cloud or third party servers). 

First of all, we need to add Spring content project dependencies. Add following two new dependencies into your build.gradle file.

compile group: 'com.github.paulcwarren', name: 'content-fs-spring-boot-starter', version: '1.1.0.M3'
compile group: 'com.github.paulcwarren', name: 'spring-content-rest-boot-starter', version: '1.1.0.M3'

Once the dependencies are added, build the project, so these files are downloaded to your machine. Now we need to configure the Spring project required configuration using @EnableFilesystemStores annotation. For this lets create a configuration class as following:

@Configuration
@EnableFilesystemStores
public class ApplicationConfig {

   @Bean
   File filesystemRoot() {
       try {
           return Files.createTempDirectory("").toFile();
       } catch (IOException ioe) {}
       return null;
   }

   @Bean
   public FileSystemResourceLoader fileSystemResourceLoader() {
       return new FileSystemResourceLoader(filesystemRoot().getAbsolutePath());
   }
}

In above code, we have added a file bean which will create a temporary directory, you can change this path to anywhere (but applications should be able to access this path). Then we have created a bean of FileSystemResourceLoader to load the resources with the absolute path. The most important part is the annotation @EnableFilesystemStores added on top of class along with @Configuration annotation. This will enable the file system and provide a temporary path to application for usage during streaming.

After the above step, create a video interface with annotation @StoreRestResource. In annotation @StoreRestResource, we have passed a path “/videos” which will be present inside resources/static and should contain at least 1 video file.

@StoreRestResource(path = "/videos")
public interface VideoStore extends Store<String> {

}

Similarly lets create an audio interface with a path as “/audios” in @StoreRestResource annotation.

@StoreRestResource(path = "/audios")
public interface AudioStore extends Store<String> {

}

After this we are done configuring Spring content project. We will not create any REST endpoints manually, Spring content will automatically create the REST apis for directory paths mentioned in @StoreRestResource annotation.

Start the application and try to access the following urls directly in browser:

http://127.0.0.1:8080/videos/video1.mp4
http://127.0.0.1:8080/audios/audio1.mp3 

You will see you are able to access videos and audios using these REST endpoints. Both media will stream automatically and browsers will start playing these. The main thing here, we are not creating any REST api manually, it’s being created by Spring content automatically.

Configure Spring Asynchronous Request Processing – Taskexecutor

When you are using StreamingResponseBody its highly recommended to configure the TaskExecutor for executing asynchronous requests in Spring MVC. In an application’s production environment, it makes sense to configure TaskExecutor to limit the threads for async processing, otherwise you can overload your server resources. Following is the sample configuration used for this example, you can modify it as per your application server’s configuration.

@Configuration
@EnableAsync
@EnableScheduling
public class AsyncConfig implements AsyncConfigurer {
   private final Logger log = LoggerFactory.getLogger(AsyncConfig.class);

   private final TaskExecutionProperties taskExecutionProperties;

   public AsyncConfig(TaskExecutionProperties taskExecutionProperties) {
       this.taskExecutionProperties = taskExecutionProperties;
   }


   @Override
   public AsyncUncaughtExceptionHandler getAsyncUncaughtExceptionHandler() {
       return new SimpleAsyncUncaughtExceptionHandler();
   }

   @Override
   @Bean(name = "taskExecutor")
   public Executor getAsyncExecutor() {
       log.debug("Create Async Task Executor");
       ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
//        executor.setCorePoolSize(taskExecutionProperties.getPool().getCoreSize());
//        executor.setMaxPoolSize(taskExecutionProperties.getPool().getMaxSize());
//        executor.setQueueCapacity(taskExecutionProperties.getPool().getQueueCapacity());
//        executor.setThreadNamePrefix(taskExecutionProperties.getThreadNamePrefix());
       executor.setCorePoolSize(1);
       executor.setMaxPoolSize(2);
       executor.setQueueCapacity(50);
       return executor;
   }

   @Bean
   protected ConcurrentTaskExecutor getTaskExecutor() {
       return new ConcurrentTaskExecutor(this.getAsyncExecutor());
   }

   @Bean
   protected WebMvcConfigurer webMvcConfigurer() {
       return new WebMvcConfigurer() {
           @Override
           public void configureAsyncSupport(AsyncSupportConfigurer configurer) {
               configurer.setTaskExecutor(getTaskExecutor());
           }
       };
   }

}

Examples Source code 

All the examples mentioned in above sections are part of a single Spring Boot project and can be run collectively using different REST endpoints. You can download the source code from github here. Feel free to use this source code for any purpose learning or commercial.

Conclusion

We have seen multiple examples for Streaming data with Spring Boot RESTful web services. Where data be raw content, JSON, CSV, file, audio, video or ZIP file. Spring Boot has all the capability to handle each one of these media types. I hope this article will help you in learning. If you want to implement authentication over some api, then you can check the detailed steps for Spring Boot security authentication here.