feat: Add option to adjust writer buffer size for query output

[m09526]

Which issue does this PR close?

• Closes Add configuration option to adjust ObjectStore's BufWriter upload size to support large file uploading to S3 #15656.

Rationale for this change

DataFusion uses object_store's BufWriter to write query results to remote object stores. Internally, this may be implemented using a remote store API to upload an object in separate chunks. For example, AWS S3 imposes a maximum count of 10,000 chunks. With a default chunk size of 10 MiB, this limits total output size to 10,000 x 10 = 100 GiB.

To allow DataFusion to write objects larger than this limit, we want to increase the buffer size. Clients expecting to write a large output can increase the chunk size via the execution option objectstore_writer_buffer_size.

What changes are included in this PR?

Allow client to set size of buffer used when DataFusion is writing output data to object_store. This is controlled via the new execution option objectstore_writer_buffer_size.

Are these changes tested?

Should all be covered by existing tests in DataFusion.

Are there any user-facing changes?

New configuration option in ExecutionOptions.
New function create_writer_with_size in datafusion::datasource::file_format::write.
Documentation added to datafusion/docs/source/user-guide/configs.md.

[alamb]

Thank you @m09526 -- in general this looks great. I have just one small API suggestion

[alamb]

If we are going to add a new API, can we please add a Builder style one here so future additional options are easier ?

So instead of

            let mut object_store_writer = create_writer_with_size(
                FileCompressionType::UNCOMPRESSED,
                &path,
                Arc::clone(&object_store),
                context
                    .session_config()
                    .options()
                    .execution
                    .objectstore_writer_buffer_size,

Something more like

            let mut object_store_writer = ObjectStoreWriterBuilder::new(
                FileCompressionType::UNCOMPRESSED,
                &path,
                Arc::clone(&object_store)
             ).with_buffer_size(
                context
                    .session_config()
                    .options()
                    .execution
                    .objectstore_writer_buffer_size,
            ).build();

[alamb]

And then maybe mark create_writer as deprecated per this page: https://datafusion.apache.org/contributor-guide/api-health.html#deprecation-guidelines (next release will be 48.0.0)

And update create_writer to use the new builder?

[m09526]

Thank you for the review @alamb, I'll get on to those improvements.

[m09526]

I've added a builder API which tries to match the requested approach.