Fetch Remote Files and Insert Data into Database with Spring Boot, Batch, and Quartz Scheduler

The image illustrates the data flow for fetching files from a remote server and inserting them into a database using Spring Boot, Spring Batch, and Quartz Scheduler.

To implement a solution where a file is fetched from a remote server and its contents are inserted into a database using Spring Boot with Spring Batch and Quartz Scheduler, follow these steps:

---

1. Setup Your Spring Boot Project

Add the necessary dependencies in pom.xml for Spring Batch, Quartz Scheduler, and any other required libraries (e.g., database drivers).

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-batch</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-quartz</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-jpa</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>mysql</groupId>
        <artifactId>mysql-connector-java</artifactId>
    </dependency>
</dependencies>

---

2. Define the Batch Configuration

Job Configuration

Create a Spring Batch configuration to define a job that reads the file, processes its content, and writes to the database.

@Configuration
@EnableBatchProcessing
public class BatchConfig {

    @Autowired
    private DataSource dataSource;

    @Bean
    public FlatFileItemReader<String> fileReader() {
        FlatFileItemReader<String> reader = new FlatFileItemReader<>();
        reader.setResource(new FileSystemResource("path/to/your/local/file.csv")); // Placeholder
        reader.setLineMapper((line, lineNumber) -> line);
        return reader;
    }

    @Bean
    public ItemProcessor<String, YourEntity> processor() {
        return line -> {
            // Parse the line and map to YourEntity
            String[] fields = line.split(",");
            return new YourEntity(fields[0], fields[1]); // Adjust based on file structure
        };
    }

    @Bean
    public JdbcBatchItemWriter<YourEntity> writer() {
        JdbcBatchItemWriter<YourEntity> writer = new JdbcBatchItemWriter<>();
        writer.setDataSource(dataSource);
        writer.setSql("INSERT INTO your_table (field1, field2) VALUES (:field1, :field2)");
        writer.setItemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>());
        return writer;
    }

    @Bean
    public Step step(StepBuilderFactory stepBuilderFactory) {
        return stepBuilderFactory.get("file-to-database-step")
                .<String, YourEntity>chunk(10)
                .reader(fileReader())
                .processor(processor())
                .writer(writer())
                .build();
    }

    @Bean
    public Job job(JobBuilderFactory jobBuilderFactory, Step step) {
        return jobBuilderFactory.get("fileToDatabaseJob")
                .start(step)
                .build();
    }
}

---

3. Fetch File from Remote Server

Use an appropriate library (e.g., Apache Commons Net, JSch, or Spring WebClient) to download the file from the remote server.

@Component
public class FileDownloader {

    public void downloadFile(String remoteUrl, String localPath) throws IOException {
        URL url = new URL(remoteUrl);
        try (InputStream in = url.openStream();
             FileOutputStream out = new FileOutputStream(localPath)) {
            byte[] buffer = new byte[1024];
            int bytesRead;
            while ((bytesRead = in.read(buffer)) != -1) {
                out.write(buffer, 0, bytesRead);
            }
        }
    }
}

---

4. Schedule the Batch Job with Quartz

Define a Quartz job to trigger the batch process periodically.

@Component
public class QuartzJob implements Job {

    @Autowired
    private JobLauncher jobLauncher;

    @Autowired
    private Job job;

    @Override
    public void execute(JobExecutionContext context) throws JobExecutionException {
        try {
            JobParameters params = new JobParametersBuilder()
                    .addLong("time", System.currentTimeMillis())
                    .toJobParameters();
            jobLauncher.run(job, params);
        } catch (Exception e) {
            throw new JobExecutionException(e);
        }
    }
}

Quartz Scheduler Configuration

Configure the Quartz Scheduler.

@Configuration
public class QuartzConfig {

    @Bean
    public JobDetail jobDetail() {
        return JobBuilder.newJob(QuartzJob.class)
                .withIdentity("fileToDatabaseJob")
                .storeDurably()
                .build();
    }

    @Bean
    public Trigger trigger(JobDetail jobDetail) {
        return TriggerBuilder.newTrigger()
                .forJob(jobDetail)
                .withIdentity("fileToDatabaseTrigger")
                .withSchedule(CronScheduleBuilder.cronSchedule("0 0/5 * * * ?")) // Runs every 5 minutes
                .build();
    }
}

---

5. Entity and Repository

Define the entity and repository for the database.

@Entity
public class YourEntity {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;
    private String field1;
    private String field2;

    // Constructors, getters, setters
}

public interface YourEntityRepository extends JpaRepository<YourEntity, Long> {}

---

6. Application Properties

Configure the application properties.

spring.datasource.url=jdbc:mysql://localhost:3306/yourdb
spring.datasource.username=youruser
spring.datasource.password=yourpassword
spring.quartz.job-store-type=jdbc

---

7. Run the Application

Start your Spring Boot application, and the Quartz scheduler will periodically fetch the file, run the batch job, and insert the data into the database.

Buy Now – Unlock Your Microservices Mastery for Only $9!

Get your copy now for just $9! and start building resilient and scalable microservices with the help of Microservices with Spring Boot 3 and Spring Cloud.